#weight-sparse transformer

'OpenAI developed a weight-sparse transformer that is far more interpretable than typical LLMs, enabling researchers to trace exact internal circuits that implement simple algorithms. While much smaller and slower than state-of-the-art models, this work could illuminate how larger models reason and fail.'